feat(rkllm): per-core NPU pinning and Ollama-compat API#4
Open
jaylfc wants to merge 3 commits intofreed-dev-llc:mainfrom
Open
feat(rkllm): per-core NPU pinning and Ollama-compat API#4jaylfc wants to merge 3 commits intofreed-dev-llc:mainfrom
jaylfc wants to merge 3 commits intofreed-dev-llc:mainfrom
Conversation
RK3588 has 3 NPU cores at ~2 TOPS each. This change lets you pin an exo node to specific cores so multiple nodes can run on the same board without contention. Set RKNN_CORE_MASK=0x1 for core 0, 0x2 for core 1, 0x4 for core 2, or 0x7 for all three. The core_mask is passed through the HTTP client to rkllama's load_model endpoint, and the engine exposes a capability descriptor with the core count and estimated TOPS so the topology manager can weigh nodes by actual capacity. Also adds a 3-node LXC test harness (scripts/test-3node-lxc.sh) that creates 3 Incus containers on one RK3588, each pinned to a single NPU core, to test distributed pipeline parallelism on consumer ARM hardware.
Tested live against rkllama on RK3588 Orange Pi 5 Plus. The Ollama compat mode uses /api/tags for model listing and /api/generate for inference, matching the NotPunchnox/rkllama server API. This is the default mode since most RK3588 deployments run rkllama. Verified: health check, model listing, and text generation all work against a live rkllama instance with qmd-query-expansion model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RKNN_CORE_MASKenv var to pin the RKLLM engine to specific RK3588 NPU cores (0x1, 0x2, 0x4 for individual cores, 0x7 for all). Useful for multi-tenant workloads on a single board./api/tagsfor model listing and/api/generatefor inference. This is the default mode since most RK3588 deployments run rkllama.get_capability_descriptor()to expose NPU core count and estimated TOPS to the topology manager.scripts/test-3node-lxc.shtest harness for multi-node testing on a single RK3588 via Incus containers.Testing
Tested on Orange Pi 5 Plus (RK3588, 16GB RAM, librknnrt 2.3.0):
/sys/kernel/debug/rknpu/loadshowing 3 coresNotes
Pipeline-parallel layer sharding is not possible with the current RKLLM SDK because
.rkllmmodels are compiled monoliths. Feature request filed at airockchip/rknn-llm#489. The core_mask feature is for multi-tenant/multi-model workloads, not model sharding.